> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/ikawrakow/ik_llama.cpp/llms.txt
> Use this file to discover all available pages before exploring further.
# Function calling
> Use OpenAI-style tool calling with any model via Jinja templates
`ik_llama.cpp` supports [OpenAI-style function calling](https://platform.openai.com/docs/guides/function-calling) through Jinja chat templates. When the server is started with `--jinja`, it parses tool definitions from incoming requests, formats them according to the model's chat template, and extracts tool calls from the model's response.
You **must** start `llama-server` with the `--jinja` flag to enable function calling. Without it, tool definitions in API requests are ignored.
## Native and generic handlers
Function calling works with all models, but the quality depends on which format handler is selected:
**Native handlers** parse tool calls using model-specific logic. They produce the most reliable results. The following model families have native support:
* Llama 3.1 / 3.2 / 3.3 (including built-in tools: `wolfram_alpha`, `web_search` / `brave_search`, `code_interpreter`)
* Qwen 2.5 and Qwen 2.5 Coder
* Hermes 2 and Hermes 3
* Mistral Nemo
* Firefunction v2
* Command R7B
* Functionary v3.1 / v3.2
* DeepSeek R1 (WIP — the model is reluctant to call tools)
**Generic handler** is used when the chat template is not recognised by any native handler. You will see `Chat format: Generic` in the server logs. Generic mode works but may consume more tokens and be less efficient than a model's native format.
## Starting the server
### Native support (no template override needed)
```bash theme={null}
llama-server --jinja -fa -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M
llama-server --jinja -fa -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K_L
llama-server --jinja -fa -hf bartowski/Llama-3.3-70B-Instruct-GGUF:Q4_K_M
```
### DeepSeek R1 (template override recommended)
The official DeepSeek R1 chat template has known issues. Use the bundled override:
```bash theme={null}
llama-server --jinja -fa \
-hf bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF:Q6_K_L \
--chat-template-file models/templates/llama-cpp-deepseek-r1.jinja
llama-server --jinja -fa \
-hf bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF:Q4_K_M \
--chat-template-file models/templates/llama-cpp-deepseek-r1.jinja
```
### Models that require a template override
Some GGUF files embed an incorrect or default (non-tool-use) template. Pass the correct template with `--chat-template-file`:
```bash theme={null}
# Functionary v3.2
llama-server --jinja -fa \
-hf bartowski/functionary-small-v3.2-GGUF:Q4_K_M \
--chat-template-file models/templates/meetkai-functionary-medium-v3.2.jinja
# Hermes 2 Pro
llama-server --jinja -fa \
-hf bartowski/Hermes-2-Pro-Llama-3-8B-GGUF:Q4_K_M \
--chat-template-file models/templates/NousResearch-Hermes-2-Pro-Llama-3-8B-tool_use.jinja
# Hermes 3
llama-server --jinja -fa \
-hf bartowski/Hermes-3-Llama-3.1-8B-GGUF:Q4_K_M \
--chat-template-file models/templates/NousResearch-Hermes-3-Llama-3.1-8B-tool_use.jinja
# Firefunction v2
llama-server --jinja -fa \
-hf bartowski/firefunction-v2-GGUF \
-hff firefunction-v2-IQ1_M.gguf \
--chat-template-file models/templates/fireworks-ai-llama-3-firefunction-v2.jinja
# Command R7B
llama-server --jinja -fa \
-hf bartowski/c4ai-command-r7b-12-2024-GGUF:Q6_K_L \
--chat-template-file models/templates/CohereForAI-c4ai-command-r7b-12-2024-tool_use.jinja
```
### Generic format models
These work out of the box with `--jinja`, using the generic handler:
```bash theme={null}
llama-server --jinja -fa -hf bartowski/phi-4-GGUF:Q4_0
llama-server --jinja -fa -hf bartowski/gemma-2-2b-it-GGUF:Q8_0
llama-server --jinja -fa -hf bartowski/c4ai-command-r-v01-GGUF:Q2_K
```
## Chat template override
If a model's embedded template is buggy or missing tool-use support, download the correct `.jinja` file and pass it with `--chat-template-file`. This avoids re-downloading the full GGUF:
```bash theme={null}
llama-server --jinja -fa \
--model /models/my-model.gguf \
--chat-template-file /models/templates/my-model-tool_use.jinja
```
To retrieve the official template from a HuggingFace repository:
```bash theme={null}
python scripts/get_chat_template.py
```
If no official `tool_use` template exists for your model, try `--chat-template chatml`. It works with many models as a fallback, though results vary.
## Making a tool call request
Use the standard OpenAI `/v1/chat/completions` endpoint with a `tools` array:
```bash theme={null}
curl http://localhost:8080/v1/chat/completions -d '{
"model": "gpt-3.5-turbo",
"tools": [
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and country/state, e.g. San Francisco, CA"
}
},
"required": ["location"]
}
}
}
],
"messages": [
{
"role": "system",
"content": "You are a helpful assistant that uses tools."
},
{
"role": "user",
"content": "What is the weather in Istanbul?"
}
]
}'
```
A successful response looks like:
```json theme={null}
{
"choices": [
{
"finish_reason": "tool",
"index": 0,
"message": {
"content": null,
"tool_calls": [
{
"name": "get_current_weather",
"arguments": "{\"location\": \"Istanbul, Turkey\"}"
}
],
"role": "assistant"
}
}
],
"model": "gpt-3.5-turbo",
"object": "chat.completion"
}
```
## KV cache and tool calling quality
Extreme KV cache quantizations (e.g. `-ctk q4_0`) can substantially degrade tool calling performance. Use `-ctk q8_0` or `-ctk q6_0` when running function-calling workloads. For very aggressive quantization below Q6\_0, add `--k-cache-hadamard` to partially recover quality.
## Verifying the active template
To confirm which template and format handler the server selected, check the `/props` endpoint after startup:
```bash theme={null}
curl http://localhost:8080/props | jq '.chat_template, .chat_template_tool_use'
```
The server logs also print the detected chat format (e.g. `Chat format: Hermes 2 Pro` or `Chat format: Generic`) at startup.